Add official EVM Block test data and execution by ceyonur · Pull Request #35 · ava-labs/strevm

ceyonur · 2025-12-04T10:31:55Z

This PR adds the test execution (from https://github.com/ava-labs/libevm/tree/main/tests) and testdata as a submodule from the repository https://github.com/ethereum/tests (at fa51c5c164 as same commit in libevm@main). It only executes BlockchainTests and LegacyTests which are inserting test blockchains (in json files) with SAE's own blockbuilder/chain (through SUT) and then executes with saexec. Tests are passing but with few limitations and hacks (see saexec/ethtests/block_test.go and comments for that file) :

1- GeneralStateTest are too slow tried to run them but CI fails see comment
2- We cannot run InvalidBlocks test because they're asserting either that block insertion is failing (which we don't have any verification yet), or they fail at execution (which we don't allow). We can revisit them once we add the verification.
3- There are some tests like bcForkStressTest and bcMultiChainTest testing multi-chains and forks with non-linear blocks and multiple possible chains. We currently don't support this in test chain builders. So these tests are skipped.
4- We don't have any execution pre-checks yet (like MaxCodeSize and intrinsic gas checks) so some tests are failing with a FAIL log in saexec. Once we have those checks we can execute those tests.
5- SAE has it's own baseFee applied to the header just before the execution, which means header's baseFee won't directly applied to the state where EVM tests expect it to be applied as it's; thus failing some state tests like balances, rewards etc. This is currently handled by faking the gas clock in insertWithHeaderBaseFee

blocks/blockstest/blocks.go

ceyonur · 2025-12-16T16:59:09Z

saexec/ethtests/block_test.go

+	// using 4.6 TGas
+	bt.skipLoad(`.*randomStatetest94.json.*`)
+
+	// TODO(cey): We cannot run invalid blocks, since we don't currently have pre-insert checks.


Most of this file is a copy-pasta from libevm. Here we specifically skip some tests that's not currently supported.

Most of this file is a copy-pasta from libevm

Why? The EEST documentation states that they are...

test fixtures (JSON) which can be consumed by any execution client to verify their implementation

...meaning that they're not even language-specific, let alone related to an implementation. I very strongly believe that we should be using these as inputs in an SAE-specific manner instead of trying to shoe-horn SAE into geth.

These were not skipped due to a forced integration between geth and sae, but rather we do not have any means of invalidating a block insertion (due to lack of validations in SAE). Maybe we can now revisit this as we have implemented a few of them, but I rather to have that in a separate PR.

from PR description:

2- We cannot run InvalidBlocks test because they're asserting either that block insertion is failing (which we don't have any verification yet), or they fail at execution (which we don't allow). We can revisit them once we add the verification.

saexec/ethtests/block_test.go

ceyonur · 2025-12-16T16:59:46Z

saexec/ethtests/block_test.go

+
+// TestExecutionSpecBlocktests runs the test fixtures from execution-spec-tests.
+func TestExecutionSpecBlocktests(t *testing.T) {
+	if !common.FileExist(executionSpecBlockchainTestDir) {


this is actually an empty file, so we don't run this test.

Then why is it included in this PR?

this is an outdated comment I forgot to remove. We now download those spec fixtures in CI (fixtures_develop.tar.gz)

saexec/ethtests/init.go

saexec/ethtests/chain_header_reader.go

saexec/ethtests/consensus_hooks.go

saexec/ethtests/sut.go

saexec/execution.go

This reverts commit 9144414.

.github/workflows/go.yml

ceyonur · 2025-12-23T14:45:47Z

saexec/ethtests/block_test_util.go

+	return validBlocks, nil
+}
+
+func insertWithHeaderBaseFee(tb testing.TB, sut *SUT, bs types.Blocks) {


this is the hack to fake the gas clock in parent by setting excess to calculated header.BaseFee

This is a useful primitive so please abstract blockstest.WithFakeBaseFee(testing.TB, *blocks.Block, [other necessary args]) *blocks.Block.

blocks/blockstest/blocks.go

ARR4N · 2026-01-03T14:54:01Z

blocks/blockstest/blocks.go

+	gen := conf.genesisSpec
+	if gen == nil {
+		gen = &core.Genesis{
+			Config:    config,
+			Timestamp: conf.timestamp,
+			Alloc:     alloc,
+		}


The default value should instead be set on conf and options.ApplyTo() might then override it. Placing defaults in the original allows options to use them; even if not something that's being done here, it's best practice.

ARR4N · 2026-01-03T14:56:50Z

.github/workflows/go.yml

+      EXECUTION_SPEC_TESTS_FILE: fixtures_develop.tar.gz
    steps:
      - uses: actions/checkout@v4
+        with:


Please put all of these in a new job, perhaps go_spec_tests, and add it to the needs of go (to make CI require it). The rationale is that the other tests are more likely to be the ones failing and we want to give that feedback as quickly as possible—placing these in a new job makes it run in parallel.

ARR4N · 2026-01-03T15:01:48Z

blocks/blockstest/chain.go

 }

+// Insert adds a block to the chain.
+func (cb *ChainBuilder) Insert(block *blocks.Block) {


NewChainBuilder() (for genesis) and ChainBuilder.NewBlock() should use Insert() so all paths to insert a block are equivalent.

ARR4N · 2026-01-03T15:14:50Z

blocks/blockstest/chain.go

+func (cb *ChainBuilder) Insert(block *blocks.Block) {
+	cb.chain = append(cb.chain, block)
+	cb.blocksByHash.Store(block.Hash(), block)
+}


Suggested change

func (cb *ChainBuilder) Insert(block *blocks.Block) {

cb.chain = append(cb.chain, block)

cb.blocksByHash.Store(block.Hash(), block)

}

func (cb *ChainBuilder) Insert(tb testing.TB, b *blocks.Block) {

tb.Helper()

if len(cb.chain) > 0 {

p := cb.Last()

require.Equalf(tb, p.Height()+1, b.Height(), "%T.Insert() non-incrementing block height", cb)

require.Equalf(tb, p.Hash(), b.ParentHash(), "%T.Insert() with mismatched parent hash", cb)

}

cb.chain = append(cb.chain, b)

cb.blocksByHash.Store(b.Hash(), b)

}

Otherwise we risk breaking internal invariants.

My initial thought was we should not fail any tests in those cases as it's intentional to insert invalid blocks (as stated here).

However it seems this should be fine. I checked that in upstream by putting a panic here and seems there is no such case https://github.com/ava-labs/libevm/blob/main/core/blockchain.go#L1532-L1533.

Anyway this change should fine for this PR as we do not enable InvalidBlocks tests yet.

However it seems this should be fine. I checked that in upstream by putting a panic here and seems there is no such case https://github.com/ava-labs/libevm/blob/main/core/blockchain.go#L1532-L1533.

On the second thought: This only checks if the given chain is ordered and linked, but it does not really check anything if it's a chain of single block (in a case we're inserting a single block). So tests actually expect a returned error (not failed test) in such cases.

As I said it won't fail in this PR, but it seems we need to return errors when we actually insert blocks (either in this blockstest package or through a prod-level chain builder)

ARR4N · 2026-01-03T15:22:31Z

blocks/blockstest/chain.go

+
+// GetHashAtHeight returns the hash of the block at the given height, and a flag indicating if it was found.
+// If the height is greater than the number of blocks in the chain, it returns an empty hash and false.
+func (cb *ChainBuilder) GetHashAtHeight(num uint64) (common.Hash, bool) {


This is only ever used to pass the hash into ChainBuilder.GetBlock() so could just be BlockByNumber(testing.TB, uint64) *blocks.Block and similarly GetNumberByHash() could just be BlockByHash(testing.TB, common.Hash) *blocks.Block.

Since they don't need to conform to any particular signature (unlike GetBlock(), which is a blocks.Source) they can do the test failures internally if the block doesn't exist.

I changed them but omitted the TB param, since we're calling these from the readerAdapter from consensus.ChainHeaderReader , which we cannot change the signature and take TB. We can set a TB for the readerAdapter when we create them but not sure if it's a good idea to carry it around.

ARR4N · 2026-01-03T19:45:50Z

saexec/ethtests/block_test.go

+	// using 4.6 TGas
+	bt.skipLoad(`.*randomStatetest94.json.*`)
+
+	// TODO(cey): We cannot run invalid blocks, since we don't currently have pre-insert checks.


Most of this file is a copy-pasta from libevm

Why? The EEST documentation states that they are...

test fixtures (JSON) which can be consumed by any execution client to verify their implementation

...meaning that they're not even language-specific, let alone related to an implementation. I very strongly believe that we should be using these as inputs in an SAE-specific manner instead of trying to shoe-horn SAE into geth.

ARR4N · 2026-01-03T19:48:24Z

saexec/ethtests/block_test_util.go

+	opts = append(opts, withGenesisSpec(gspec))
+
+	// Wrap the original engine within the beacon-engine
+	engine := beacon.New(ethash.NewFaker())


Invoking the Beacon chain is IMO strongly suggestive of this not being an appropriate approach.

indeed, this is why we require #72. Otherwise we will always need to comply with Ethereum rules and specs/fixtures for testing these.

My understanding from stand-up yesterday was that #72 was to allow for testing of coreth/subnet-evm variants, but not for SAE/libevm as they would just use upstream spec tests. What I mean (here too) is that the JSON can be parsed to build and run blocks directly against the saexec.Executor.

While #72 will be crucial for coreth/subnet-evm, bu SAE is also not 1-1 compliant with those JSON fixtures (baseFee, missing blob handling, beacon etc). We're currently patching those through upstream engines and before/after hooks with upstream-like handlings (difficulty, coinbase, beacon etc). In fact we should generate those fixtures without running any of these (and skip some specs/fixtures like beacon/difficulty/blobs i.e stuff we don't really support/have)

I'm not sure what you mean by parsing JSON. We do parse them, create blocks from those JSONs and run blocks directly against the executor (here). If you mean modifying JSON that will be generating fixtures (as described in #72).

ARR4N · 2026-01-12T09:43:08Z

saexec/ethtests/block_test.go

+
+// TestExecutionSpecBlocktests runs the test fixtures from execution-spec-tests.
+func TestExecutionSpecBlocktests(t *testing.T) {
+	if !common.FileExist(executionSpecBlockchainTestDir) {


Then why is it included in this PR?

ARR4N · 2026-01-12T09:45:55Z

saexec/ethtests/block_test_util.go

+	return validBlocks, nil
+}
+
+func insertWithHeaderBaseFee(tb testing.TB, sut *SUT, bs types.Blocks) {


This is a useful primitive so please abstract blockstest.WithFakeBaseFee(testing.TB, *blocks.Block, [other necessary args]) *blocks.Block.

This reverts commit 1636b3d.

ceyonur changed the base branch from main to ceyonur/init-eth-tests December 4, 2025 10:42

ceyonur force-pushed the ceyonur/eth-tests branch from 29bcee8 to 728dd98 Compare December 16, 2025 15:42

ceyonur changed the base branch from ceyonur/init-eth-tests to main December 16, 2025 15:43

ceyonur added 2 commits December 16, 2025 19:26

integrate eth tests

cd6ecad

readd submodule

6ecc876

ceyonur force-pushed the ceyonur/eth-tests branch from 728dd98 to 6ecc876 Compare December 16, 2025 16:44

ceyonur added 4 commits December 16, 2025 19:50

recursive checkout for ci

b3d828c

add gencodec tool

33f61ea

increase timeout

3a2fdf3

run only short tests

b318591

ceyonur commented Dec 16, 2025

View reviewed changes

ceyonur added 2 commits December 17, 2025 00:36

retry running all tests

9144414

Revert "retry running all tests"

cff1884

This reverts commit 9144414.

ceyonur changed the title ~~Ceyonur/eth tests~~ Add go-ethereum/tests testdata Dec 17, 2025

ceyonur changed the title ~~Add go-ethereum/tests testdata~~ Add official EVM testdata Dec 17, 2025

ceyonur self-assigned this Dec 17, 2025

ceyonur linked an issue Dec 17, 2025 that may be closed by this pull request

Run official EVM tests against SAE #27

Open

ceyonur changed the title ~~Add official EVM testdata~~ Add official EVM Block test data and execution Dec 17, 2025

ceyonur commented Dec 17, 2025

View reviewed changes

.github/workflows/go.yml Outdated Show resolved Hide resolved

ceyonur mentioned this pull request Dec 17, 2025

Run official EVM tests against SAE #27

Open

ceyonur added 3 commits December 23, 2025 16:20

insert with faked base fee

f3e2dc3

fix jank init

822b7b1

resolve todos

fc5083f

ceyonur commented Dec 23, 2025

View reviewed changes

ceyonur and others added 6 commits December 23, 2025 17:47

Merge branch 'main' into ceyonur/eth-tests

647863b

linters

2514095

add eest tests

047cc4e

Merge branch 'main' into ceyonur/eth-tests

1255d15

add NewGenesisFromSpec

21586c6

comments

2007d73

ceyonur added 12 commits December 29, 2025 13:12

comments

55a58f2

fix target param

ae5adc5

uncommitted file

308dd5d

add checksum verificaiton

e29cb8c

Merge branch 'main' into ceyonur/eth-tests

ad3d299

linters

0c4cf09

merge new genesis helpers

d912c37

Merge branch 'main' into ceyonur/eth-tests

a2fafb1

fix license header

1787110

add checksums file

ac16882

fix paths

b8a5a9c

move checksums file

f94a0bb

ceyonur commented Jan 2, 2026

View reviewed changes

blocks/blockstest/blocks.go Show resolved Hide resolved

ceyonur marked this pull request as ready for review January 2, 2026 13:34

ARR4N reviewed Jan 12, 2026

View reviewed changes

ceyonur added 11 commits January 13, 2026 16:01

add WithFakeBaseFee

0662874

change default overriding

3be8b19

move spec testing to new step

b12f7ae

use insert

29dfc1a

change block fetching functions

b2a5466

conditionally add the hook

283d305

Merge branch 'main' into ceyonur/eth-tests

95208fa

use worst case boundaries in Insert

2484133

comments

dbb0487

debug tests

1636b3d

Revert "debug tests"

33ee46d

This reverts commit 1636b3d.

-func (cb *ChainBuilder) Insert(block *blocks.Block) {
-	cb.chain = append(cb.chain, block)
-	cb.blocksByHash.Store(block.Hash(), block)
-}
+func (cb *ChainBuilder) Insert(tb testing.TB, b *blocks.Block) {
+	tb.Helper()
+	if len(cb.chain) > 0 {
+		p := cb.Last()
+		require.Equalf(tb, p.Height()+1, b.Height(), "%T.Insert() non-incrementing block height", cb)
+		require.Equalf(tb, p.Hash(), b.ParentHash(), "%T.Insert() with mismatched parent hash", cb)
+	}
+	cb.chain = append(cb.chain, b)
+	cb.blocksByHash.Store(b.Hash(), b)
+}

Conversation

ceyonur commented Dec 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ceyonur Jan 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ceyonur Jan 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ceyonur Jan 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

ceyonur commented Dec 4, 2025 •

edited

Loading

ceyonur Jan 13, 2026 •

edited

Loading

ceyonur Jan 13, 2026 •

edited

Loading

ceyonur Jan 13, 2026 •

edited

Loading